Skip to content

Python Job/Task Queues

Landscape Overview

Redis dominates as broker. PostgreSQL is gaining ground (fewer moving parts). asyncio support is now table stakes for new libraries. Celery remains dominant by inertia but its complexity keeps pushing users toward alternatives.

Major Options

Celery — The Incumbent

  • Repo: https://github.com/celery/celery (~28k stars)
  • Brokers: RabbitMQ, Redis, SQS, Zookeeper
  • Status: Active. v5.6.x current, v6.0 in planning.
  • Batteries-included: Canvas workflows (chains/chords/groups), Celery Beat (periodic tasks), broad broker support.
  • Heavy dependency tree (kombu, billiard, vine). Operational complexity is well-documented.
  • Async story remains weak.
  • Critical memory leaks fixed in 5.6. Python 3.14 support added.

Post-mortems worth reading:
- Celery in production: 3 more years of fixing bugs
- Two years with Celery in production

Dramatiq — The Better Celery

  • Repo: https://github.com/Bogdanp/dramatiq (~4.5k stars)
  • Brokers: RabbitMQ, Redis
  • Status: Active. v2.1.x.
  • Designed explicitly as a simpler, more reliable Celery. Fewer footguns.
  • Built-in rate limiting, retries, pipelines. dramatiq-workflow adds chains/groups.
  • Good middle ground between Celery’s feature breadth and RQ’s simplicity.

RQ (Redis Queue) — The Simple One

  • Repo: https://github.com/rq/rq (~10k stars)
  • Brokers: Redis, Valkey
  • Status: Active. v2.x.
  • Maximally simple: queue.enqueue(func, args). Built-in scheduling since v2.5.
  • Sync workers only — no async support.
  • Good when Redis is already present and needs are straightforward.

Huey — The Lightweight One

  • Repo: https://github.com/coleifer/huey (~5.2k stars)
  • Brokers: Redis, SQLite, in-memory, file-system
  • Status: Active. v2.6.x. Python 3.14 support.
  • Minimal dependencies. Periodic tasks, pipelines, locking, signals.
  • SQLite backend useful for dev/small deployments.
  • By Charles Leifer (peewee author).

Async-Native Options

TaskIQ — The Modern Async Ecosystem

  • Repo: https://github.com/taskiq-python/taskiq (~2k stars)
  • Brokers: Redis, RabbitMQ, NATS, Kafka (via plugins)
  • Status: Active. Updated March 2026.
  • Fully async. Modular broker/backend via plugins.
  • First-class FastAPI and AioHTTP integration. Dependency injection. Strong typing.
  • Most complete “modern async Celery alternative.”

SAQ (Simple Async Queues)

  • Repo: https://github.com/tobymao/saq (~1k stars)
  • Brokers: Redis, PostgreSQL
  • Status: Active. Jan 2026 release.
  • Inspired by ARQ, faster (<5ms latency). Built-in web UI.
  • Litestar integration available (litestar-saq).

Streaq — The Redis Streams Newcomer

  • Repo: https://github.com/tastyware/streaq (~500 stars)
  • Brokers: Redis (via Redis Streams, not lists)
  • Status: Active. v6.0.x.
  • Redis Streams = better reliability than list-based queues.
  • asyncio + Trio support (via anyio). Fully typed. Graceful shutdown.
  • Python 3.14 free-threaded support. Spiritual successor to ARQ.

ARQ — Deprecated

  • Repo: https://github.com/python-arq/arq (~2.2k stars)
  • Status: Maintenance-only. Effectively dead.
  • By Samuel Colvin (pydantic). Users should migrate to SAQ or Streaq.

PostgreSQL-Based (Transactional Enqueuing)

Procrastinate — The PostgreSQL-Native Choice

  • Repo: https://github.com/procrastinate-org/procrastinate (~2k stars)
  • Brokers: PostgreSQL only
  • Status: Active.
  • Uses LISTEN/NOTIFY + FOR UPDATE SKIP LOCKED. No Redis/RabbitMQ needed.
  • Tasks enqueued transactionally with your data. Django integration.
  • Both sync and async. Periodic tasks, retries, locks.
  • Follows the “use the database you already have” philosophy (cf. Rails’ Solid Queue).

pq

Niche / Specialized

Library Repo Notes
TaskTiger closeio/tasktiger (~1.4k stars) Redis. Unique: task deduplication, batch queues, per-customer subqueues. Battle-tested at Close.io.
WakaQ wakatime/wakaq (~300 stars) Redis. “Super minimal Celery.” Priority queues, cron. Deliberately minimal — used in production at WakaTime.
Hatchet hatchet-dev/hatchet (~4k stars) PostgreSQL / managed cloud. Not just a queue — DAG orchestration + durable execution. Python/TS/Go SDKs. Competes more with Temporal than Celery. YC W24.
LiteQ ddreamboy/liteq SQLite-only, zero external dependencies. Async + sync, priority queues, cron scheduling, retries, FastAPI integration. True zero-infra option. Very young project (Feb 2026).

Dead / Unmaintained

Decision Matrix

Need Recommendation
Feature-complete, battle-tested Celery
Simpler Celery with fewer bugs Dramatiq
Dead simple, sync RQ or Huey
Async-native, plugin ecosystem TaskIQ
Async-native, lightweight SAQ or Streaq
No Redis, just PostgreSQL Procrastinate
DAG orchestration / durable execution Hatchet

References

#asyncio #redis #async #celery #job-queue #task-queue

Page last modified: 2026-04-01 19:31:08